Improvements in computer hardware and technology coupled with the multiplication of connected mobile electronic devices has amplified interest in developing artificial intelligence and solutions for task automatization, outcome prediction, information classification and learning from experience, resulting in the field of machine learning. Machine learning, closely related to data mining, computational statistics and optimization, explores the study and construction of algorithms that can learn from and make predictions based on data.
The field of machine learning has evolved extensively in the last decade, giving rise to self-driving cars, speech recognition, image recognition, effective web search, personalization, and understanding of the human genome, among others.
Machine learning algorithms (MLA) may generally be divided into broad categories such as supervised learning, unsupervised learning and reinforcement learning. Supervised learning consists in presenting a machine learning algorithm with training data consisting of inputs and outputs labelled by assessors, where the objective is to train the machine learning algorithm such that it learns a general rule for mapping inputs to outputs. Unsupervised learning consists in presenting the machine learning algorithm with unlabeled data, where the objective is for the machine learning algorithm to find a structure or hidden patterns in the data. Reinforcement learning consists in having an algorithm evolving in a dynamic environment without providing the algorithm with labeled data or corrections.
Recent advances in the field have also produced active learning, a form of semi-supervised learning, born from situations where unlabeled data is abundant, but where labeling data can be expensive. In such situations, learning algorithms may query assessors for labels when needed, therefore iteratively improving their models while possibly requiring less data.
Learning to rank (LTR) or machine learned ranking (MLR) is the application of machine learning in the construction of ranking models for information retrieval, natural language processing and data mining. Generally, a system may maintain a collection of documents, where a ranking model may rank documents responsive to a query, and return the most relevant documents. The ranking model may have been previously trained on sample documents. However, the sheer number of documents available on the Internet combined with its continuous growth makes labeling not only difficult, but requires a lot of computational and monetary resources, as it is often performed by human assessors.
U.S. Pat. No. 8,713,023 issued Apr. 29, 2014 to Cormack et al. teaches systems and methods for classifying electronic information or documents into a number of classes and subclasses through an active learning algorithm. Such document classification systems are easily scalable for large document collections, require less manpower and can be employed on a single computer, thus requiring fewer resources. Furthermore, the classification systems and methods described can be used for any pattern recognition or classification effort in a wide variety of fields, including electronic discovery in legal proceedings.
U.S. Pat. No. 7,844,567 issued Nov. 30, 2010 to He et al. teaches a system and method for selecting a training sample from a sample set. The method comprises determining proximities between all data samples in a set of the data samples, forming edges between the data samples as a function of the proximities, computing weights for the edges as a function of the proximities, selecting a plurality of the data samples as a function of the weights to form a subset of the data samples, and storing the subset of the data samples.
U.S. Patent Publication No. 2012/0109860 A1 by Xu et al. teaches that training data is used by learning-to-rank algorithms for formulating ranking algorithms. The training data can be initially provided by human judges, and then modeled in light of user click-through data to detect probable ranking errors. The probable ranking errors are provided to the original human judges, who can refine the training data in light of this information.
For the foregoing reasons, there is a need for methods and systems for selecting potentially erroneously ranked documents by a machine learning algorithm.